Boot Camp for Cognitive Systems

The Defense Advanced Research Projects Agency (DARPA) has implemented a program to build the first instance of a complete cognitive agent. The program, called Personalized Assistant that Learns (PAL), is expected to yield new cognitive technology of significant value not only to the military, but also to the business and academic sectors. (Gunning 2004) With traditional engineering projects, evaluation can be done in a straightforward manner determining if the documented requirements of the system have been met. Agent-based capabilities and other network centric capabilities (e.g., web services) complicate matters because the environment that they will operate under constantly changes. Add to that complication, the ability to learn new capabilities, and testing whether or not a new agent is ready to be deployed becomes a problem beyond the current state of art and practice. In this paper an initial experiment design is discussed as well as a description of a broader approach for evaluation in transitioning cognitive systems that learn into an operational environment.